An interactive imaging experience includes an environment in which an interactive display is affected by the motion of human bodies, or the like. A camera, or set of cameras, detects a number of features of human bodies or other objects disposed before the camera, such as their silhouettes, hands, head, and direction of motion, and determines how these features geometrically relate to the visual display. For example, a user interacting before a front-projected display casts a shadow on an optional display medium such as a projection screen, or the like. The interactive imaging system is capable of aligning the camera's detection of the silhouette of the human body with the shadow of the human body. This geometric alignment creates a natural mapping for controlling elements in the visual display. Persons of all ages can likely recall an experience of playing with their shadows and can thus understand that their motion in front of a source of bright light will produce a shadow whose motion behaves exactly as expected. This experience is capitalized upon in an interactive imaging experience.
Body motion capture is the analysis of a body in movement whereby the process includes recording a motion event and translating it into mathematical terms. Motion capture technologies that exist in the prior art include electromechanical, electromagnetic, and optical tracking systems. The electromechanical systems require the user to wear a bodysuit containing measuring devices at fixed points. The electromagnetic systems approach requires electromagnetic sensors to be worn upon a user body at specific locations. The optical motion capture technique uses multiple cameras and requires markers attached at various body locations of the user. Each of these approaches requires some sort of body suit and/or markers to be placed on a user's body. In addition to this need for specialized equipment, the equipment itself is expensive, restrictive, and limiting to the user. Markerless motion capture, on the other hand, allows motion capture without the need for such equipment and markers attached to a body.
There are many uses for motion capture technologies. Key among those is the use of human motion capture for use in computer-generated virtual characters or objects, such as virtual reality interactions for example. Such a use often requires various recordings of human motion, action, and interaction and then processing and using that data in virtual characters or objects.
Automatic interaction of virtual characters with real persons in a machine-sensed environment is now a very popular form of entertainment. For example, in Dance Dance Revolution® (DDR), (see http://www.konami.com/, generally, and see http://www.konamistyle.com/b2c_kden/b2c/init.do, specifically), a popular video game produced by Konami, three-dimensional virtual characters or objects or two-dimensional cartoon characters dance along with players in the context of a song with beat-driven motion instructions. The game includes a dance pad with arrow panels, providing instructions to a player as to whether to move up, down, left, or right.
An example of virtual characters that respond to a player instead of mimicking the player is the EyeToy® by Nam Tai (see http://www.us.playstation.com/eyetoy.aspx), a color digital camera device for the PlayStation2 by Sony. The EyeToy® device is similar to a web camera and uses computer vision to process images taken by the camera. In one EyeToy® boxing game, players “box” with a virtual character that reacts to punches and “hits” back. EyeToy® incorporates a form of gesture recognition in order to manipulate the virtual boxer in response to punches thrown by the real player.
While Dance Dance Revolution® and the EyeToy® are compelling for entertaining people through body-to-body interactions, each has an obvious synthetic presence. The virtual bodies in each game could not be mistaken for real characters in the real world. Furthermore, the vision-based (as opposed to button-control-based DDR®) EyeToy® boxing game cannot react to real human motion in a convincing way due to deficiencies with the degrees of freedom and the overall level of detail in the captured motion of the player.
While these patents and other previous systems and methods have attempted to solve the above mentioned problems, none have provided a system and method for enabling meaningful body-to-body interaction with virtual video-based characters or objects. Thus, a need exists for such a system and method.